Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Draft: Windows: Diagnose msys openssl fork issue #13276

Closed
wants to merge 1 commit into from
Closed

Draft: Windows: Diagnose msys openssl fork issue #13276

wants to merge 1 commit into from

Conversation

sunjayBhatia
Copy link
Member

DO NOT MERGE

See failed build: https://dev.azure.com/cncf/4684fb3d-0389-4e0b-8251-221942316e06/_apis/build/builds/52196/logs/126

error:

e is 65537 (0x010001)
      1 [main] openssl 125 child_info_fork::abort: \??\C:\tools\msys64\usr\bin\msys-crypto-1.1.dll: Loaded to different address: parent(0x58B980000) != child(0x400000)
Can't open good_cert.pem for reading, No such file or directory
34359738384:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('good_cert.pem','r')
34359738384:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:
unable to load certificate
Failed bash -c "PYTHONPATH=$(dirname C:/botcode/w/bazel-out/x64_windows-opt/bin/test/extensions/transport_sockets/tls/handshaker_test.exe.runfiles/envoy/test/extensions/transport_sockets/tls/ocsp/gen_unittest_ocsp_data.sh) C:/botcode/w/bazel-out/x64_windows-opt/bin/test/extensions/transport_sockets/tls/handshaker_test.exe.runfiles/envoy/test/extensions/transport_sockets/tls/ocsp/gen_unittest_ocsp_data.sh "
[2020-09-25 18:47:59.960][2168][critical][assert] [test/test_common/environment.cc:413] assert failure: false.

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]

@sunjayBhatia
Copy link
Member Author

looks like disabling usosvc fixes the issue (update orchestrator service)

this PR is required to fix this defect: envoyproxy/envoy-build-tools#100

@sunjayBhatia sunjayBhatia mentioned this pull request Sep 29, 2020
Merged
@sunjayBhatia
Copy link
Member Author

ran a test that utilizes msys2 openssl: https://dev.azure.com/cncf/4684fb3d-0389-4e0b-8251-221942316e06/_apis/build/builds/52525/logs/16

succeeds when usosvc is stopped

@sunjayBhatia sunjayBhatia deleted the windows-diagnose-openssl-msys-fork-issue branch September 30, 2020 00:28
@sunjayBhatia sunjayBhatia restored the windows-diagnose-openssl-msys-fork-issue branch October 2, 2020 19:31
@sunjayBhatia sunjayBhatia reopened this Oct 2, 2020
@sunjayBhatia
Windows: Diagnose msys openssl fork issue
8e8bc61 
See failed build: https://dev.azure.com/cncf/4684fb3d-0389-4e0b-8251-221942316e06/_apis/build/builds/52196/logs/126

error:

```
e is 65537 (0x010001)
      1 [main] openssl 125 child_info_fork::abort: \??\C:\tools\msys64\usr\bin\msys-crypto-1.1.dll: Loaded to different address: parent(0x58B980000) != child(0x400000)
Can't open good_cert.pem for reading, No such file or directory
34359738384:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('good_cert.pem','r')
34359738384:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:
unable to load certificate
Failed bash -c "PYTHONPATH=$(dirname C:/botcode/w/bazel-out/x64_windows-opt/bin/test/extensions/transport_sockets/tls/handshaker_test.exe.runfiles/envoy/test/extensions/transport_sockets/tls/ocsp/gen_unittest_ocsp_data.sh) C:/botcode/w/bazel-out/x64_windows-opt/bin/test/extensions/transport_sockets/tls/handshaker_test.exe.runfiles/envoy/test/extensions/transport_sockets/tls/ocsp/gen_unittest_ocsp_data.sh "
[2020-09-25 18:47:59.960][2168][critical][assert] [test/test_common/environment.cc:413] assert failure: false.
```

Signed-off-by: Sunjay Bhatia <[email protected]>
@sunjayBhatia sunjayBhatia mentioned this pull request Oct 6, 2020
Merged
@sunjayBhatia
Copy link
Member Author

ran a test that utilizes msys2 openssl: https://dev.azure.com/cncf/4684fb3d-0389-4e0b-8251-221942316e06/_apis/build/builds/52525/logs/16

succeeds when usosvc is stopped

Turns out this was wrong, the invocation of powershell was enough to slow down the script and make the flake occurrence less frequent but it still exists

@sunjayBhatia
Copy link
Member Author

It looks like possibly an issue as described here, not entirely sure https://cygwin.com/faq.html#faq.using.bloda

@sunjayBhatia
Copy link
Member Author

Running msys2 autorebase.bat when building the build container image does not help

This flake only occurs on RBE, we have not been able to repro it locally on our dev VM or in a basic GCP Windows server VM

@sunjayBhatia
Copy link
Member Author

Also attempting to build and use the bssl tool as an openssl tool equivalent from Boring SSL will not suffice, it has a much more limited set of capabilities as far as I can tell

@wrowe
Copy link
Contributor

wrowe commented Oct 12, 2020

Thanks for all the research. With the complexity of TSL1.3 features and QUIC, the most flexible build host tool is probably needed. What are your thoughts, should we always pick up a modern openssl and build the same tool for every host, or deploy the win32 flavor of openssl at provisioning time?

@sunjayBhatia
Copy link
Member Author

sunjayBhatia commented Oct 12, 2020 •
edited
Loading

Thanks for all the research. With the complexity of TSL1.3 features and QUIC, the most flexible build host tool is probably needed. What are your thoughts, should we always pick up a modern openssl and build the same tool for every host, or deploy the win32 flavor of openssl at provisioning time?

Given that OpenSSL isn't bazelified already, adding another foreign_cc build to get the openssl.exe tool may not get accepted? I tried mirroring what chocolatey does and installing with the 64 bit OpenSSL installer from here, and it works well, I'll run the Envoy tests building a build container with that. We could maybe even build the openssl.exe tool in the build container build process, but using the installer is quicker.

@sunjayBhatia sunjayBhatia mentioned this pull request Oct 13, 2020
Closed
@sunjayBhatia
Copy link
Member Author

Thanks for all the research. With the complexity of TSL1.3 features and QUIC, the most flexible build host tool is probably needed. What are your thoughts, should we always pick up a modern openssl and build the same tool for every host, or deploy the win32 flavor of openssl at provisioning time?

Given that OpenSSL isn't bazelified already, adding another foreign_cc build to get the openssl.exe tool may not get accepted? I tried mirroring what chocolatey does and installing with the 64 bit OpenSSL installer from here, and it works well, I'll run the Envoy tests building a build container with that. We could maybe even build the openssl.exe tool in the build container build process, but using the installer is quicker.

Installing standalone OpenSSL worked, see envoyproxy/envoy-build-tools#107 for the build container update coming soon

@sunjayBhatia sunjayBhatia mentioned this pull request Oct 22, 2020
Merged
@sunjayBhatia
Copy link
Member Author

Fixed by #13702

@sunjayBhatia sunjayBhatia deleted the windows-diagnose-openssl-msys-fork-issue branch October 22, 2020 23:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@wrowe wrowe

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sunjayBhatia @wrowe